Thing You Should Know

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring batch-oriented workflows.

Airflow's extensible Python framework enables you to build workflows connecting with virtually any technology. A web interface helps manage the state of your workflows. Airflow is deployable in many ways, varying from a single process on your laptop to a distributed setup to support even the largest workflows.

Workflows as Code

The main characteristic of Airflow workflows is that all workflows are defined in Python code. "Workflows as code" serves several purposes:

Monitoring DAGs in the Airflow Web UI

The Airflow Web UI serves as a central cockpit for managing and monitoring workflows. It lists all registered DAGs, showing their schedules, owners, execution history, active statuses, and execution timings:

Understanding Task State Colors on the Screen

When you monitor or test a DAG in the Airflow UI (such as in Grid View, Graph View, or Tree View), Airflow represents each task instance as a small block or node. As the Airflow Scheduler and Worker execute the tasks, the screen updates dynamically, changing the box color to visually indicate the real-time execution state:

The DAG — Airflow's Fundamental Abstraction

In Airflow, a DAG (Directed Acyclic Graph) is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies. A DAG is defined in a Python script, which represents the DAGs structure (tasks and their dependencies) as code.

graph LR
    subgraph "DAG: etl_sales_pipeline"
        A["extract_from_api"] --> B["validate_schema"]
        A --> C["extract_from_db"]
        B --> D["transform_data"]
        C --> D
        D --> E["load_to_warehouse"]
        E --> F["run_data_quality"]
        F --> G["notify_stakeholders"]
    end
    style A fill:#4CAF50,stroke:#388E3C,color:#fff
    style B fill:#FF9800,stroke:#F57C00,color:#fff
    style C fill:#4CAF50,stroke:#388E3C,color:#fff
    style D fill:#2196F3,stroke:#1976D2,color:#fff
    style E fill:#9C27B0,stroke:#7B1FA2,color:#fff
    style F fill:#F44336,stroke:#D32F2F,color:#fff
    style G fill:#607D8B,stroke:#455A64,color:#fff

Key Properties of a DAG

• Directed — Each dependency has a clear direction. Task A runs before Task B.
• Acyclic — There are no circular dependencies. You cannot have A → B → C → A.
• Graph — Tasks are nodes, dependencies are edges. This structure enables parallel execution.

Defining a DAG in Python

from airflow.sdk import DAG, dag
from airflow.providers.standard.operators.python import PythonOperator
from datetime import datetime, timedelta

# Method 1: Context Manager (Most Common)
with DAG(
    dag_id="etl_sales_pipeline",
    description="Daily ETL pipeline for sales data",
    schedule="0 6 * * *",          # Run daily at 6:00 AM UTC
    start_date=datetime(2024, 1, 1),
    catchup=False,                  # Don't backfill historical runs
    tags=["production", "sales"],
    default_args={
        "owner": "data-engineering",
        "retries": 3,
        "retry_delay": timedelta(minutes=5),
        "email_on_failure": True,
        "email": ["data-team@company.com"],
    },
) as dag:

    extract = PythonOperator(
        task_id="extract_from_api",
        python_callable=extract_sales_data,
    )

    transform = PythonOperator(
        task_id="transform_data",
        python_callable=transform_sales_data,
    )

    load = PythonOperator(
        task_id="load_to_warehouse",
        python_callable=load_to_snowflake,
    )

    # Define execution order
    extract >> transform >> load

# Method 2: TaskFlow API with @dag Decorator (Airflow 2.0+)
from airflow.sdk import dag, task
from datetime import datetime

@dag(
    schedule="@daily",
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=["production"],
)
def etl_sales_pipeline():

    @task()
    def extract():
        \"\"\"Extract data from source systems.\"\"\"
        return {"data": [1, 2, 3]}

    @task()
    def transform(raw_data: dict):
        \"\"\"Clean and transform raw data.\"\"\"
        return {"transformed": raw_data["data"]}

    @task()
    def load(transformed_data: dict):
        \"\"\"Load data into warehouse.\"\"\"
        print(f"Loading {len(transformed_data['transformed'])} records")

    # Dependencies are inferred from function calls
    raw = extract()
    transformed = transform(raw)
    load(transformed)

etl_sales_pipeline()

Core Terminology

What Airflow is NOT

Understanding what Airflow is not is equally important for making the right architectural decisions:

This content is reserved for Premium Members.